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A remark about orthogonal matching pursuit 
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Abstract 



In this note, we investigate the theoretical properties of Orthogonal 
Matching Pursuit (OMP), a class of decoder to recover sparse signal in 
compressed sensing. In particular, we show that the OMP decoder can 
give (p, q) instance optimality for a large class of encoders with 1 < p < 
g < 2 and {p,q) 7^ (2,2). We also show that, if the encoding matrix is 
drawn from an appropriate distribution, then the OMP decoder is (2, 2) 
^ . instance optimal in probability. 

in ■ 1 Introduction 

>. 

CO , We consider a signal x e M'^ where N is large and denote by Y,/^ the set of 

ON 

O 

m 



fc-sparse vectors, i.e., 

Sfc := {x e R^ : #supp(a;) < k}, 



in 

^^ , where supp(x) is the set of i for which x^ ^ and i^A is the number of elements 

2"^ ' in the set A. Given a norm || • \\x on R^, we set 



X. 

H and 



I3kix) := argminllx - z\\x, 



(7kix)x ■= \\x - Pkix)\\x, 



and call I3k{x) and ak{x)x as the the best k-term approximation and the best 
k-term approximation error, respectively. 

In Compressed Sensing theory, the information we gather about x can be 
described by 

y = $a; 

where ^ is an nxN matrix. To recover x from y, we use a decoder A : M" — > M.^ 
and denote x* := A(y) = A($a;). The x* can be considered as an approximation 
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of X. Following Cohen, Dahmen and DeVore, we say that a pair ($, A) is 
instance optimality of order k with constant C for the norm X if it satisfies 

\\x-A{^x)\\x<Cak{x)x, foran.TeM^. (1) 

In general, one chooses X as £p (quasi-)norm where p > 0. To state conveniently, 
throughout of this paper, we use the subscript p to denote the ip norm. Using 
the notation of [T^, we say that a pair ($,A) is {q,p) instance optimality of 
order k with constant C if 

\\x - A($x)|L < C-^^^, for ah x € K^. 

The theoretical analysis of instance optimality is presented in [4] . 

We next introduce a class of encoding matrix. Following Candes and Tao, 
we say that the matrix $ satisfies the Restricted Isometry Property (RIP) of 
order k and constant Sk & (0, 1) if 

{l-Sk)\\x\\l<\\^x\\l<{l + 6k)\\x\\l (2) 

holds for all x e E^. Throughout the rest of the paper, using the notation 
of [m, we say that the matrix $ satisfies RIP(fc,(5) if 5k < S. 

In the past, one investigated the instance optimality of ii minimization, 
which is given by 

AJ(y) := argmin||x||i, subject to \\^x — y\\2 < e. 

X 

In [2], Candes improved on the work of Candes, Romberg and Tao 0] and 
showed that 

\\Al{<^x) ~ x\\2 < Coak{x)i/V^ + Cie 

provided $ satisfies RIP(2fc, \/2 ~ 1). The result implies that, in the noise-free 
case, i.e., e = 0, ($,A]') is (2,1) instance optimality of order fc if $ satisfies 
RIP(2fc, ^/2 — 1). In [3], Saab and Yilmaz extended the results to ip decoder, 
where < p < 1, which is defined by 

Ap(y) := argmin||a;||p subject to ||$a; — y\\2 < e, 

X 

and showed that ($, A^) is {2,p) instance optimality of order A: if $ satisfies 
some RIP condition (see [S]). Independently, Foucart and Lai [B] also proved 
that ($, Ap) is (2,p) instance optimality under other sufficient conditions. 

In compressed sensing, an alternative decoder is Orthogonal Matching Pur- 
suit (OMP). The major advantages of OMP are its ease of implementation and 
potentially faster than A| (see [71 [TU1[TT| ) . However, so far, very few theoretical 
results about the instance optimality of OMP are known. The aim of the note is 
the investigation of the instance optimality property of OMP decoder. To state 
conveniently, we use OMPj\/ to denote the OMP decoder with M iterations (see 
Algorithm 1). Then combining the methods developed by Cohen, Dahmen and 
DeVore [4] and the result obtained by Zhang [13], we can prove the following 
result, which is the main result of this note: 



Theorem 1. Suppose that 1 < p < q < 2 and < S < 1. We furthermore 
suppose that p ^ 2. Let $ be any matrix which satisfies RIP condition RIP(L, 6) 
and 6k + (1 + 6)6ak < S, where a := [16 + 155] and L :^ k {N/kf^'^'''\ Then 
for any signal x and any permutation e with ||ej|2 < e, the solution x* := 
0MP2(„_i)fc($x + e) obeys 

\\^*-AU < Co^j^ + c.k^/'^-'^h, (3) 

where Co = I + C\ + (2a)i/9-i/2 and Ci = {2ay/'^-^/^{2{l + S){VTT+206 + 
1) + 1)- 



Algorithm 1 OMPA/(y) 

Input: encoding matrix $, the vector y, maximum allowed sparsity M 

Output: the x* . 

Initialize: r° = y, c° = 0, A° = 0,^ = 0. 

while £ < M do 

match: h^ = $^r^ 

identity: A^+i = A^ U {argmaXj|/i^(.7)|} 

update: c^'^^ — argmin Hy — $z||2 

e = e + i 

end while 

X* ^ c^' 

By setting [q^p) = (2, 1) and e = in Theorem [TJ we obtain the foUowing 
Corollary: 

Corollary 1. Suppose that $ satisfies the RIP condition 52k + (1 + '^)'52Qfe 1^ ^■ 
Then 

llOMP2(,_i)fe($a;) - x||2 < C2<yk{x)i/Vk, 



where a= [16+15(5] and C2 = 2(1 + (5)(VTT+2(M + 1) + 3. 

Remark 1. Theorem [T] implies that ($,OMP) is {q,p) instance optimality of 
order k provided <& satisfies RIP condition of order k{N/k)^^^^P. Note that 
n X N matrix <1> can have RIP of order fc if A: = 0{n/ log{N/n)). Then, 
for OMP decoder. {q,p) instance optimality can be achieved at the price of 
0{k{N/ky~^^Plog{N/k)) measm-ements. As shown in [4] (Theorem 7.3), the 
number of measurements is optimal up to a constant. 

Remark 2. No recover method can improve the term k^^'^~^/^e on the right 
side of ([3]) for arbitrary perturbations e. To see why this true, suppose that 
we take a: S S^ and we know in advance the support of x, i.e., Tq ~ supp(a:). 



Using this additional information, as shown in [3], Least-Square has the best 
performance to recover x. Set y = ^x + e. Then 

^* ^ U^To^To)-''^'T„y. on To, 

1 0, elsewhere . 

A simple observation is that {x* — x)t<: = and 

[X* -x)t„ = ($^„$To)~'$Toe- 

Here, we use Tg to denote the complement of T and XTg to denote the vector 
which agrees with x on Tq and has all components equal to zero on Tq . Then 
Holder inequality implies that 

where the equality holds for some non-zero permutation e. Since $ satisfies RIP 
condition of order L, 

||(*^„$To)-'<i>^„e|l2«|l$^„e||2«e, 

which implies that ||.t* — x\\q w fc^'''^^'^e for some non-zero permutation e 
provided a; € Efc. 

We next consider the (2, 2) instance optimality. As pointed out in [4], to ob- 
tain (2, 2) instance optimality with order fc = 1, one has to require the number of 
measurements is 0{N), which is not what wc hope. Hence, instance optimality 
in probability is the proper formulation in £2- We let i7 be a probability space 
with probability measure P and let $ = $(aj),aj G 51, be an n x iV random 
matrix and suppose A(a;) is a corresponding family of decoders. Wc say that 
($(w), A(a;)) is (2, 2) instance optimality in probability of order k with constant 
Cif 

||x-A($.T)|l2<Cafc(.T)2 

holds with high probability for this particular x. To state our results, we first 
introduce two properties that the random matrix $ should satisfy (see [4]). 

Definition 1. We say that the nx N random matrix $ satisfies RIP of order k 
with constants Sk and probability 1 — e if there is a set Oq C O with P{ilo) > 1 — e 
such that for all uj G Hq the matrix $(a;) satisfies 

(1 - 4)||a;||2 < ll*Ma;|l2 < (1 + '5fe)||a;||i for aU x e S^. 

Definition 2. We say that the random matrix $ has the boundedncss property 
with constant C and probability 1 — e if for each x G M^ there is a set JIq (x) C Jl 
with P(i7o(a;)) > 1 — e such that for all uj £ nQ{x), 

mu^)x\\l<C\\x\\l (4) 



The key thing in Definition [5] is that the set of w's where (JH) holds depends 
on X. The both properties above have been shown for random matrices of the 
Gaussian or Bernoulli type (see [T]). Now we can state the result of instance 
optimality with probability for £2 norm. 

Theorem 2. Suppose < S < 1 and a := [16 + 15(5] . Assume that $ is 
a random matrix which satisfies the RIP condition dk + (1 + S)Sak < S with 
probability 1 — e, and also satisfies the boundedness property with constant C 
and probability 1 — e. Then for each x G M^, there exists a set ^{x) C Vl with 
P{Vl{x)) > 1 - 2e such that for all uj € ^{x) and <I> = <i>(a;), 

||a;* - x\\2 < C3crfc(a;)2 

where x* := OMP(„_i)fc($a;) and C3 = 1 + ^C{1 + S){1 + Vll + 20(5). 

Remark 3. According to Thcorcm[21 OMP can achieve (2,2) instance optimal- 
ity in probability of order k after at least 15fc steps. Suppose that $ satisfy the 
conditions in Theorem [21 Then, the following question is interesting: what is 
the minimal value of Cq for which OMP can achieve (2,2) instance optimality 
in probability of order k after cgk iterations? 

2 The proofs of Theorem [1] and Theorem [2] 

We first recall a result obtained by T. Zhang, which plays an important role in 
our analysis. 

Theorem 3. ( fl^ ) Let x e R^ andO < S <1. If the RIP condition S\\s\\o + 
(1 + (5)(5Q,||g||p < S holds, then when s ^ (a — 1)||.t||o, we have 

\\'^x*-y\\j<{n + 20S)\\<^x~y\\l 

where x* = 0MP^(2/) and a = [16 + 15(5] . 

We also need the following lemmas: 

Lemma 1. Let $ be any matrix which satisfies RIP(i,(5) with S < 1 and 

Then for any z G M^ and 1 < p < 2 we have 

\\^z\\2<VT+S{\\zh+\\z\\p/k'^'^-'/'-). 

Proof. Let Ti denote the set of indices of the L largest entries of z, T2 the next 
L largest, and so on. The last set T^ defined this way may have less than L 
elements. Set Tq := U^^j^i- Then 

||$Z||2 < ||$2:Ti||2+|1«'2tJ12 + --- + 11$^tJ12 

< Vl + SiWzT.h + \\ZT2\\2 + --- + II^Tjb). (5) 



Since for any i G Tj+i and i' G Tj, we have \zi\ < \zii\, which imphes that 

Then we obtain that 

Noting h — I < N/L, we obtain that 

Then © and © give 

\\<^zh<VTTsi\\zh + \\z\yk'/^-'/'). 

n 

Lemma 2. ( ^) Set r := l/p — 1/q and suppose r > 0. Then 

<yk{z), < \\z\yk\ 

We now have all ingredients to prove our conclusion: 

Proof of Theorem[l[ We first consider the (2,p) case with 1 < p < 2. Taking 
X := P2k{x) in Theorem |31 we obtain that 



|l$a;*-y||2 < Vlf + 20(5||$/32fc(a;) - *a- - e||2 

< x/TTT2M(||$/32fc(x)-$a-||2 + ||e||2), 

where x* := 0MP2(Q_i)fc($a; + e). Noting that #supp(x* - P2k{x)) < 2ak and 
l/Vl - S2ak < \/l + S, we have 

||.^*-/32fe(a:)||2 

< VT+SW^x* ~ <Pl32k{x)h 

< VT+S mx* - yh + \\^l32k{x) - j/Hz) 

< ^/m(Vll + 205 + l)(||*^2fc(s) - <^x\\2 + ||e||2) 

< {l + S)iVnT20S+l){a2kix)2 + a2k{x)p/k^/P-^^^ + \\e\\2), (7) 

where the last inequality uses Lemma [1] We now consider the term (J2k{x)2 
which appears on the right side of ([7]). Lemma [5] gives 

a2fc(x)2 = akix - (3k{x))2 < \\x - /3fc(x)||p/fci/P-i/2 = akix)p/k'/P-^^\ (8) 

Combining ([7]) and (|8]), we obtain that 

Ik* - I32k{x)\\2 < 2(1 + S)Wn + 205 + l){akix)p/k^^P-'/^ + \\e\\2). (9) 



Equations (|5]) and (O imply that 

Ik*-^ll2 

< 02k{x)2 + \\X* - P2k{x)h 

< (2(1 + <5)(V11 + 20s + 1) + 1) K(x)p/A:i/P-i/2 ^ jigji^)^ (10) 

Wc next consider the general case. We first recall that Holder inequality, 
which says 

(|ai|" + • • • + |a,ry/"(|6ir + • • • + I^/cD'/^ > |ai&i| + • • • + \akbk\ 

provided 1/u+l/i' = 1 and u,v > where a,b E K'^. Then, by Holder inequality, 
when 1 < q < 2, we have 

((|5^|9)2A/ + • • • + (IfefeD' /')'^' • fci-«/2 > I61I'' + • • • + |6fc|«, 

which implies that 

\\bh > Jfyj. (11) 

It follows from expressions ([8]) and ([T0| that the condition 



||a;* - /32fc(a;)||2 < ik'* - -^Ib + o-2fe(a;)2 

< C"K(x)p/fci/P-i/2 + ||e||2) + afc(x)p/fci/P-V2 

< (C" + lK(.T)p/fci/p-i/2 + c'||e|l2, 



where C" = 2(1 + 6){Vll + 205 + 1) + 1. Note that #supp(2;* - /32k{x)) < 2ak. 
The inequality (jlip provides the bound 

Then, combining inequalities above, we arrive at 
\\x*-/32k{x)\U < (2a)i/*-i/2(C'+l)fci/''-i/Pafc(a;)p+C"(2afc)i/'?-i/2||g||2. (12) 

From Lemma [5] and expression (fT^ . we obtain the relation 

II "^ II 
\\x -x\\g 

< \\X ~ I32k{x)\\q + \\X* - (52k{x)\\q = CF2k{x)q + \\x* - (i2k{x)\\q 

< afe(x)p/fci/P-i/9 + (2a)i/9-i/2fci/?-i/P(c' + l)ak{x)p + C'{2aky/'^-^^^e\\2 

< (1 + (2a)i/'/-i/2(c' + 1))-^^^ + C'(2a)i/^-i/2fci/*-i/2e, 

where C" = 2(1 + 5){y/ll + 2Q5 + 1) + 1. The conclusion follows. D 



Proof of Theorem [^ We build the proof on the ideas of Cohen, Dahmen and 
DeVore [4]. We give the fuU proof for completeness. 
Using the triangle inequality, we have 

\\x - x*\\2 <\\x- /3fc(x)|l2 + \\l3k{x) ~ x*||2 - afc(x)2 + ||/3fe(x) - a;*||2. (13) 

Let 57o and 51(x — f3k{x)) be, respectively, the set in the definition of RIP in 
probability and the set in the definition of boundedness in probability for the 
vector X — (ik{x). We set fi' := ^io n fi(a; — /3fe(a;)). A simple observation is 
P{Vt') > 1 - 2e. Also, Theorem [3] implies that 

\\y - ^x*\\2 < Vn + WsWy - $/3fc(x)|l2. 



Noting that l/\/l — 52ak < vl + J, for any a; e 17', we obtain that 

\\h{x)-x*h < \Mh{x) - a:*)||2/Vl - ^afc < VlT~5mh{x) - x*)\\ 

< VI + S{\\y - $.tt||2 + \\y - '^x*\\2) 

< Vl + ^(1 + Vll + 20S)\\<i>x ~ $/3fe(a;)|J2 



2 



< VcVTT6il + Vll + 205)\\x - (3k{x) 

cVTTs(i + Vn + 20S)<Jk{x)2. 



2 



Here, the first inequality uses the RIP and ^supp{f3k{x) — x*) < ak and the last 
inequality uses the boundedness property in probability for x — Pk{x). Combin- 
ing ()13p and the equation above, we have 

\\X - X*||2 < (l + ^/C{1+5){1 + Vll+205)) (Tk{x)2. 

D 
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